Red Hat Enterprise Linux 7 Troubleshooting

Boot Troubleshooting

Module Topics

Boot Process Overview
Repairing Common Boot Issues
Repairing File System Issues at Boot
Repairing Boot Loader Issues

Boot Process Overview

Boot process stages:
1. Hardware (BIOS/UEFI)
2. Boot loader (grub2)
3. kernel and initramfs
4. systemd
To reboot a system, use systemctl reboot.
To power down a system, use systemctl poweroff.
To switch to a new target at runtime, use systemctl isolate desired.target.
To query the default target, use systemctl get-default.
To set the default target, use systemctl set-default.
To select a different target at boot, use systemd.unit= on the kernel command line.

Boot Process Overview

Boot Process Details

The machine is powered on. The system firmware (UEFI or BIOS) runs a Power-On Self-Test (POST), and starts to initialize some of the hardware.
Configuration: Use the system BIOS/UEFI configuration screens. To reach these screens, press a certain key or key combination, such as F2, during the early part of the boot process.
The system firmware searches for a bootable device, either configured in the UEFI boot firmware or by searching for a Master Boot Record (MBR) on all disks, in the order configured in the BIOS.
Configuration: Use the system BIOS/UEFI configuration screens.
The system firmware reads a boot loader from disk, then passes control of the system to the boot loader. On a Red Hat Enterprise Linux 7 system, this is typically grub2.
Configuration: Use grub2-install.
The boot loader loads its configuration from disk, and presents a menu of configurations to boot.
Configuration: Use /etc/grub.d/, /etc/default/grub, and if you are not using manual configuration, you can also use /boot/grub2/grub.cfg.
After the user makes a choice (or a timeout occurs), the boot loader loads the configured kernel and initramfs from disk and places them in memory. On Red Hat Enterprise Linux 7, initramfs contains an entire usable system.
Configuration: Use /etc/dracut.conf.

== m06p03_boot_process_details_1

Modern computer systems are complex combinations of hardware and software. Starting from an undefined, powered-down state to a running system with a (graphical) login prompt requires a large number of pieces of hardware and software to work together. The following list gives a high-level overview of the tasks involved for a physical x86_64 system booting Red Hat Enterprise Linux 7. The list for x86_64 virtual machines is roughly the same, but some of the hardware-specific steps are handled in software by the hypervisor.

The machine is powered on. The system firmware (either modern UEFI or more old-fashioned BIOS) runs a Power-On Self-Test (POST), and starts to initialize some of the hardware.
Configuration: Use the system BIOS/UEFI configuration screens. These screens are typically reached by pressing a certain key or key combination, such as F2, during the early part of the boot process.
The system firmware searches for a bootable device, either configured in the UEFI boot firmware or by searching for a Master Boot Record (MBR) on all disks, in the order configured in the BIOS.
Configuration: Use the system BIOS/UEFI configuration screens. These screens are typically reached by pressing a certain key or key combination, such as F2, during the early part of the boot process.
The system firmware reads a boot loader from disk, then passes control of the system to the boot loader. On a Red Hat Enterprise Linux 7 system, this will typically be grub2.
Configuration: Use grub2-install.
The boot loader loads its configuration from disk, and presents the user with a menu of possible configurations to boot.
Configuration: Use /etc/grub.d/, /etc/default/grub, and if you are not using manual configuration, you can also use /boot/grub2/grub.cfg.
After the user makes a choice (or an automatic timeout occurs), the boot loader loads the configured kernel and initramfs from disk and places them in memory. An initramfs is a gzip-ed cpio archive containing kernel modules for all hardware required at boot, for init scripts, and more. On Red Hat Enterprise Linux 7, initramfs contains an entire usable system.
Configuration: Use /etc/dracut.conf.

Boot Process Overview

Boot Process Details

The boot loader hands control of the system to the kernel, passing in any options specified on the kernel command line in the boot loader, and the location of initramfs in memory.
Configuration: Use /etc/grub.d/, /etc/default/grub, and if you are not using manual configuration, you can also use /boot/grub2/grub.cfg.
The kernel initializes all hardware for which it can find a driver in initramfs, then executes /sbin/init from initramfs as PID 1. On Red Hat Enterprise Linux initramfs contains a working copy of systemd as /sbin/init, as well as a udev daemon.
Configuration: Use the init= command-line parameter.
The systemd instance from initramfs executes all units for initrd.target, which is the target. This includes mounting the root file system on /sysroot.
Configuration: Use /etc/fstab.
The kernel root file system is switched (pivoted) from the initramfs root file system to the system root file system that was previously mounted on /sysroot. Then systemd re-executes itself using the copy of systemd installed on the system.
systemd looks for a default target, either passed in from the kernel command line or configured on the system, and then starts (and stops) units to comply with the configuration for that target, which solves dependencies between units automatically.
Configuration: Use /etc/systemd/system/default.target or /etc/systemd/system/.

== m06p04_boot_process_details_2

The boot loader hands control of the system over to the kernel, passing in any options specified on the kernel command line in the boot loader, and the location of initramfs in memory.
Configuration: Use /etc/grub.d/, /etc/default/grub, and if you are not using manual configuration, you can also use /boot/grub2/grub.cfg.
The kernel initializes all hardware for which it can find a driver in initramfs, then executes /sbin/init from initramfs as PID 1. On Red Hat Enterprise Linux initramfs contains a working copy of systemd as /sbin/init, as well as a udev daemon.
Configuration: Use the init= command-line parameter.
The systemd instance from initramfs executes all units for initrd.target, which is the target. This includes mounting the actual root file system on /sysroot.
Configuration: Use /etc/fstab.
The kernel root file system is switched (pivoted) from the initramfs root file system to the system root file system that was previously mounted on /sysroot. Then systemd re-executes itself using the copy of systemd installed on the system.
systemd looks for a default target, either passed in from the kernel command line or configured on the system, and then starts (and stops) units to comply with the configuration for that target, which solves dependencies between units automatically. In its essence, a systemd target is a set of units that should be activated to reach a desired system state. These targets typically include at least a text-based login or a graphical login screen being spawned.
Configuration: Use /etc/systemd/system/default.target or /etc/systemd/system/.

Boot Process Overview

Boot, Reboot, and Shut Down

Use systemctl to power off or reboot a running system from the command line.
Use systemctl poweroff to stop all running services, unmount all file systems (or remount them as read-only when they cannot be unmounted), and then power down the system.
Use systemctl reboot to stop all running services, unmount all file systems, and then reboot the system.

systemctl halt and halt are also available to stop the system, but unlike their poweroff equivalents, these commands do not power off the system; they bring a system down to a point where it is safe to manually power it off.

== m06p05_boot_process_boot_re_shut

To power off or reboot a running system from the command line, administrators can use the systemctl command.

systemctl poweroff stops all running services, unmounts all file systems (or remounts them as read-only when they cannot be unmounted), and then powers down the system.

systemctl reboot stops all running services, unmounts all file systems, and then reboots the system.

For the ease of backward compatibility, the poweroff and reboot commands still exist, but in Red Hat Enterprise Linux 7 they are symbolic links to the systemctl tool.

Boot Process Overview

Selecting a `systemd` Target
Target	Purpose
`graphical.target`	Supports multiple users with graphical and text-based logins.
`multi-user.target`	Supports multiple users with text-based logins only.
`rescue.target`	Provides an `sulogin` prompt after completing basic system initialization.
`emergency.target`	Provides an `sulogin` prompt after completing an `initramfs` pivot and mounting (read only) on the system root.

A target can be part of another target. You can view these dependencies from the command line:
```
[root@server1 ~]# systemctl list-dependencies graphical.target | grep target
```

To see an overview of all available targets:

[root@server1 ~]# systemctl list-units --type=target --all

To see an overview of all targets installed on disk:

[root@server1 ~]# systemctl list-unit-files --type=target --all

Boot Process Overview

Selecting a Target at Runtime

Use systemctl isolate on a running system to switch to a different target.
```
[root@server1 ~]# systemctl isolate multi-user.target
```
Isolating a target stops all services not required by that target (and its dependencies), and starts any required services that have not yet been started.

Not all targets can be isolated. Only targets that have AllowIsolate=yes set in their unit files can be isolated. For example, graphical.target can be isolated, but cryptsetup.target cannot.

Boot Process Overview

Setting a Default Target

The systemctl tool comes with two commands to manage the default.target symbolic link: get-default and set-default.

[root@server1 ~]# systemctl get-default
multi-user.target
[root@server1 ~]# systemctl set-default graphical.target
rm '/etc/systemd/system/default.target'
ln -s '/usr/lib/systemd/system/graphical.target' '/etc/systemd/system/default.target'
[root@server1 ~]# systemctl get-default
graphical.target

Boot Process Overview

Selecting a Different Target at Boot Time

To select a different target at boot time, append systemd.unit= to the kernel command line from the boot loader.
- This change affects only a single boot, making it a useful troubleshooting tool.

Example: Boot the system into a rescue shell

systemd.unit=rescue.target

How to Select a Different Target for Red Hat Enterprise Linux 7 Systems

Boot or reboot the system.
Interrupt the boot loader menu countdown by pressing any key.
Move the cursor to the entry you want to start.
Press e to edit the current entry.
Move the cursor to the line that starts with linux16. This is the kernel command line.
Append systemd.unit=desired.target to the kernel command line.
Press Ctrl+x to boot with these changes.

References

bootup(7), dracut.bootup(7), systemd.target(5), systemd.special(7), sulogin(8), and systemctl(1) man pages
GNU GRUB Manual: info grub2

== m06p09_boot_process_select_diff_targ

To select a different target at boot time, a special option can be appended to the kernel command line from the boot loader: systemd.unit=.

For example, to boot the system into a rescue shell, where you can make configuration changes with minimal services running, append the following from the interactive boot loader menu before starting.

This configuration change only affects a single boot, making it a useful tool for troubleshooting the boot process.

To use this method of selecting a different target, use the following procedure for Red Hat Enterprise Linux 7 systems:

Boot or reboot the system.
Interrupt the boot loader menu countdown by pressing any key.
Move the cursor to the entry you want to start.
Press e to edit the current entry.
Move the cursor to the line that starts with linux16. This is the kernel command line.
Append systemd.unit=desired.target to the kernel command line.
Press Ctrl+x to boot with these changes.

Repairing Common Boot Issues

Use systemd.unit=rescue.target on the kernel command line to interrupt the boot process before control is handed over from initramfs. The system is mounted (read-only) under /sysroot.
Use journalctl to filter for specific boots with the -b option.
Use debug-shell.service to get an automatic root shell early in the boot sequence.

Repairing Common Boot Issues

Recovering the Root Password

On Red Hat Enterprise Linux 7, you can pause the scripts that run from initramfs at certain points, provide a root shell, and then continue when that shell exits. While this is mostly meant for debugging, you can also use this technique to recover a lost root password:

Reboot the system.
Interrupt the boot loader countdown by pressing any key.
Move the cursor to the entry that you want to boot.
Press e to edit the selected entry.
Move the cursor to the kernel command line.
Append systemd.unit=rescue.target.
Press Ctrl+x to boot with the changes.

At this point, a root shell is presented, with the root file system for the actual system mounted as read-only on /sysroot.

SELinux is not yet enabled, so any new files being created will not have an SELinux context. Some tools (such as passwd) first create a new file, then move it in place of the file they are intended to edit, effectively creating a new file without an SELinux context.

== m06p11_recover_root_pwd_1

One task that every system administrator should be able to accomplish is recovering a lost root password. If the administrator is still logged in, either as an unprivileged user but with full sudo access, or as root, this task is trivial. When the administrator is not logged in, this task becomes slightly more involved.

A number of methods exist to set a new root password. A system administrator can, for example, boot the system using a live CD, mount the root file system from there, and edit /etc/shadow. In this section, we will explore a method that does not require the use of external media.

On Red Hat Enterprise Linux 6; and earlier, an administrator could boot the system into runlevel 1, and be presented with a root prompt. The closest analogs to runlevel 1 on a Red Hat Enterprise Linux 7 machine are rescue.target and emergency.target, both of which require the root password to log in.

On Red Hat Enterprise Linux 7, it is possible to pause the scripts that run from initramfs at certain points, provide a root shell, and then continue when that shell exits. While this is mostly meant for debugging, you can also use this technique to recover a lost root password:

Reboot the system.
Interrupt the boot loader countdown by pressing any key.
Move the cursor to the entry that you want to boot.
Press e to edit the selected entry.
Move the cursor to the kernel command line (the line that starts with linux16).
Append systemd.unit=rescue.target (this will break just before control is handed from initramfs to the actual system).
The initramfs prompt shows up on the last console specified on the kernel command line.
Press Ctrl+x to boot with the changes.

Pre-built images may place multiple console= arguments to the kernel to support a wide array of implementation scenarios. The caveat with systemd.unit=rescue.target is that while many of the kernel messages are sent to all consoles, the prompt ultimately uses whichever console is last. If you do not get your prompt, you may want to temporarily re-order the console= arguments.

At this point, a root shell is presented, with the root file system for the actual system mounted as read-only on /sysroot.

SELinux is not yet enabled at this point, so any new files being created will not have an SELinux context assigned to them. Keep in mind that some tools (such as passwd) first create a new file, then move it in place of the file they are intended to edit, effectively creating a new file without an SELinux context.

Repairing Common Boot Issues

Recovering the Root Password

To recover the root password from this point:

Remount /sysroot as read-write.

switch_root:/# mount -oremount,rw /sysroot

Switch into a chroot jail, where /sysroot is treated as the root of the file system tree.
```
switch_root:/# chroot /sysroot
```
Set a new root password:
```
sh-4.2# passwd root
```
Make sure that all unlabeled files (including /etc/shadow) get relabeled during boot.
```
sh-4.2# touch /.autorelabel
```
Type exit twice.

Repairing Common Boot Issues

Using journalctl

Make sure that persistent journald logging is enabled:

[root@server1 ~]# mkdir -p -m2775 /var/log/journal
[root@server1 ~]# chown :systemd-journal /var/log/journal
[root@server1 ~]# killall -USR1 systemd-journald

Use the -b option with journalctl to view all messages rated as an error or worse from the previous boot:
```
[root@server1 ~]# journalctl -b-1 -p err
```

Repairing Common Boot Issues

Diagnose and Repair systemd Boot Issues

Early Debug Shell: Run systemctl enable debug-shell.service to spawn a root shell on TTY9 (Ctrl+Alt+F9) early in the boot sequence.
Disable debug-shell.service when you are done debugging. It leaves an unauthenticated root shell open to anyone with local console access.
Emergency and Rescue Targets: Append either systemd.unit=rescue.target or systemd.unit=emergency.target to the kernel command line from the boot loader to spawn into a special rescue or emergency shell instead of starting normally.
Stuck Jobs: Use systemctl list-jobs to inspect the current job list. Any jobs listed as running must complete before the jobs listed as waiting can continue.

References

dracut.cmdline(7), systemd-journald(8), journalctl(1), sushell(8), and systemctl(1) man pages
/usr/lib/systemd/system/debug-shell.service

== m06p14_diagnose_repair_systemd

If there are problems while services are starting, there are a couple of tools that can help system administrators with debugging and/or troubleshooting.

Early Debug Shell

By running systemctl enable debug-shell.service, a root shell is spawned on TTY9 (Ctrl+Alt+F9) early during the boot sequence. This shell is automatically logged in as root, so that an administrator can use some of the other debugging tools while the system is still booting.

Do not forget to disable debug-shell.service when you are done debugging; it leaves an unauthenticated root shell open to anyone with local console access.

Emergency and Rescue Targets

By appending either systemd.unit=rescue.target or systemd.unit=emergency.target to the kernel command line from the boot loader, the system spawns into a special rescue or emergency shell instead of starting normally. Both of these shells require the root password. emergency.target keeps the root file system mounted as read-only, while rescue.target waits for sysinit.target to complete first so that more of the system will be initialized (e.g., logging, file systems, etc.).

These shells can be used to fix any issues that prevent the system from booting normally; for example, a dependency loop between services, or an incorrect entry in /etc/fstab. Exiting from these shells will continue with the regular boot process.

Stuck Jobs

During startup, systemd spawns a number of jobs. If some of these jobs cannot complete, they block other jobs from running. To inspect the current job list, an administrator can use the systemctl list-jobs command. Any jobs listed as running must complete before the jobs listed as waiting can continue.

Repairing File System Issues at Boot

systemd displays an emergency shell in most cases dealing with file system issues.

emergency.target can also be used to diagnose and fix file system issues.

Problem Result

Problem	Result
Corrupt file system	`systemd` attempts an `fsck`. If the problem is too serious for an automatic fix, the user is prompted to run `fsck` manually from an emergency shell.
Non-existent device/UUID referenced in `/etc/fstab`	`systemd` waits for a set amount of time for the device to become available. If the device does not become available, the user is dropped to an emergency shell after the timeout.
Non-existent mount point in `/etc/fstab`	`systemd` creates the mount point if possible; otherwise, it drops to an emergency shell.
Incorrect mount option specified in `/etc/fstab`	The user is dropped to an emergency shell.

Corrupt file system

systemd attempts an fsck. If the problem is too serious for an automatic fix, the user is prompted to run fsck manually from an emergency shell.

Non-existent device/UUID referenced in /etc/fstab

systemd waits for a set amount of time for the device to become available. If the device does not become available, the user is dropped to an emergency shell after the timeout.

Non-existent mount point in /etc/fstab

systemd creates the mount point if possible; otherwise, it drops to an emergency shell.

Incorrect mount option specified in /etc/fstab

The user is dropped to an emergency shell.

When using the automatic recovery shell during file system issues, remember to issue a systemctl daemon-reload after editing /etc/fstab. Without this reload, systemd continues using the old version.

Reference

systemd-fsck(8), systemd-fstab-generator(3), and systemd.mount(5) man pages

== m06p15_repair_file_system

systemd displays an emergency shell in most cases dealing with file system issues.
emergency.target can also be used to diagnose and fix file system issues.

After completing this section, you should be able to repair file system issues during boot.

Errors in /etc/fstab and corrupt file systems can stop a system from booting. In most cases, systemd continues to boot after a timeout, or drops to an emergency repair shell that requires the root password.

The following table lists some common errors and their results.

In all cases, an administrator can also use emergency.target to diagnose and fix the issue, since no file systems will be mounted before the emergency shell is displayed.

When using the automatic recovery shell during file system issues, do not forget to issue a systemctl daemon-reload after editing /etc/fstab. Without this reload, systemd continues using the old version.

Repairing Boot Loader Issues

Use e and Ctrl+x to edit boot loader entries in memory, then boot.
Use grub2-mkconfig > /boot/grub2/grub.cfg to regenerate the boot loader configuration.
Use grub2-install to reinstall the boot loader.
grub2 can be used on both BIOS and UEFI systems, and supports booting almost any operating system that runs on modern hardware.
grub2-mkconfig looks at /etc/default/grub for options, and then it uses a set of scripts in /etc/grub.d/ to generate a configuration file.
To make permanent changes to the boot loader configuration, edit the configuration files listed previously, and then run the following command:
```
[root@server1 ~]# grub2-mkconfig > /boot/grub2/grub.cfg
```

== m06p16_repair_boot_loader

Use e and Ctrl+x to edit boot loader entries in memory, then boot.
Use grub2-mkconfig > /boot/grub2/grub.cfg to regenerate the boot loader configuration.
Use grub2-install to reinstall the boot loader.

After completing this section, you should be able to fix boot loader issues.

The boot loader used by default on Red Hat Enterprise Linux 7 is grub2, the second major version of the GRand Unified Bootloader.

grub2 can be used to boot on both BIOS and UEFI systems, and supports booting almost any operating system that runs on modern hardware.

The main configuration file for grub2 is /boot/grub2/grub.cfg, but administrators are not supposed to edit this file directly. Instead, a tool called grub2-mkconfig is used to generate that configuration using a set of different configuration files, and the list of installed kernels.

grub2-mkconfig looks at /etc/default/grub for options such as the default menu timeout and the kernel command line to use, and then it uses a set of scripts in /etc/grub.d/ to generate a configuration file.

To make permanent changes to the boot loader configuration, an administrator needs to edit the configuration files listed previously, and then run the following command…

In cases where major changes were made, an administrator might prefer to run this command without the redirection so that the results can be inspected first.

Repairing Boot Loader Issues

Important Directives

Bootable entries are encoded inside menuentry blocks.
linux16 and initrd16 lines point to the kernel to be loaded from disk and the initramfs files to be loaded.
Use Tab completion to find these files during interactive editing at boot.
set root lines inside those blocks point to the file system from which grub2 should load the kernel and initramfs files.
- The syntax is harddrive,partition
- hd0 is the first hard drive in the system
- hd1 is the second, etc.
- Partitions are indicated as msdos1 for the first MBR partition, or gpt1 for the first GPT partition on that drive.

== m06p17_important_directives

To troubleshoot a broken grub2 configuration, an administrator needs to understand the syntax of /boot/grub2/grub.cfg first. Actual bootable entries are encoded inside menuentry blocks. In these blocks, linux16 and initrd16 lines point to the kernel to be loaded from disk (along with the kernel command line) and the initramfs files to be loaded. During interactive editing at boot, you can use Tab completion to find these files.

The set root lines inside those blocks do not point to the root file system for the Red Hat Enterprise Linux 7 system, but instead point to the file system from which grub2 should load the kernel and initramfs files. The syntax is harddrive,partition, where hd0 is the first hard drive in the system, hd1 is the second, etc. The partitions are indicated as msdos1 for the first MBR partition, or gpt1 for the first GPT partition on that drive.

Repairing Boot Loader Issues

Reinstalling the Boot Loader

Use grub2-install to reinstall the boot loader.
On BIOS systems, provide the disk where grub2 should be installed in the MBR as an argument.
On UEFI systems, no arguments are necessary when the EFI system partition is mounted on /boot/efi.

References

GNU GRUB Manual: info grub2 and info grub2-install
Working with the Grub 2 Boot Loader

Module Completion

Nice job!

Click the button below to complete this module of the course: